General Matrix Multiplication (GEMM) is the process of multiplying two matrices. Dot-product is a computation performed in GEMM. Dot-product specifically involves multiplying two vectors. A vector is a set of values, which are organized in the form of an array. An array is a single row or single column matrix, i.e., a one-dimensional matrix.
Suppose that {right arrow over (a)} is a vector (a1, a2, a3) and {right arrow over (b)} is another vector (b1, b2, b3). In the accompanying drawings, a vector is also denoted as a letter with an arrow above the letter). The dot-product of {right arrow over (a)} and {right arrow over (b)} is shown as {right arrow over (a)}·{right arrow over (b)} and has the value (a1*b1)+(a2*b2)+(a3*b3). In a specific example, suppose that {right arrow over (a)} vector (0, 3, −7) and {right arrow over (b)} is vector (2, 3, 1). The dot-product of {right arrow over (a)} and {right arrow over (b)} is {right arrow over (a)}·{right arrow over (b)}=(0*2)+(3*3)+(−7*1)=2.
The multiplication computation in a dot-product is referred to as a “product.” For example, a1*b1 is a product. The addition computation in a dot-product is referred to as an “accumulation.” For example, the addition of products a1*b1 and a2*b2 is an accumulation. Thus, a dot-product involves several product computations and several accumulation computations.
The vectors in a dot-product can be of any size. Vectors containing hundreds or thousands of values are not uncommon. Furthermore, the values in the vectors can be of any precision. For example, a value in a vector can be 4, −3.01, 0.0000000005023, and the like. Furthermore, in a dot-product computation, a value of any precision in one vector can participate in a product operation with another value of any precision from the other vector. A product obtained in computing a dot-product can be of any precision and might accumulate with another product of any precision.
Deep learning neural networks, also referred to as Deep Neural Networks (DNN) are a type of neural networks. The configuring and training of DNNs employs dot-product computations. Over the course of the training of a DNN, many dot-products have to be computed at each iteration or cycle of training. Because the DNNs can include thousands if not millions of nodes, the scale of the dot-product computations can include vectors comprising thousands or even millions of values, many of which can be precise to any number of decimal places, e.g., the fifth, fifteenth, fiftieth decimal place or even more.